Skip to content

About

A Go package for working with URLs.

Topics

Resources

Stars

1 star

Watchers

1 watching

Forks

Repository files navigation

hq-lib-url-go

made with go go reference license maintenance open issues closed issues

hq-lib-url-go is a Go package for working with URLs.

Resources

Features

  • URL extraction: longest-match patterns find fully qualified URLs, scheme-less hosts, email addresses, and relative paths, with functional options and exported pattern blocks.
  • PSL-aware parsing: parser decomposes hosts into subdomain, SLD, and TLD against the embedded Public Suffix List, including multi-label and private suffixes, with label-aware matching.
  • Normalization: lowercase scheme and host, default-port removal, dot-segment removal, and sorted query parameters through URL.Normalize.
  • Scheme inference: InferScheme chooses http for local targets and https for everything else.
  • Typed URL values: parser.URL embeds net/url.URL and exposes the decomposed Domain, while Apex returns the registrable domain.

Installation

To install hq-lib-url-go, run the following command in your Go project:

go get github.com/hueristiq/hq-lib-url-go@latest

Requires Go 1.26.0 or later.

Usage

The examples import the packages under the hqgourl-prefixed aliases.

Quick start

The root package covers the common cases with the default parser and extractor.

package main

import (
	"fmt"
	"log"

	hqgourl "github.com/hueristiq/hq-lib-url-go"
)

func main() {
	parsed, err := hqgourl.Parse("https://example.com/path?q=1")
	if err != nil {
		log.Fatal(err)
	}

	matches, err := hqgourl.Extract("see https://example.com/a and https://example.org/b")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(parsed.String())
	fmt.Println(matches)
}

Extraction

The extractor package scans text and pulls out URLs with a composed regular expression. Extract is the one-call entry point. The pattern is compiled once per extractor and cached.

package main

import (
	"fmt"
	"log"

	hqgourlextractor "github.com/hueristiq/hq-lib-url-go/extractor"
)

func main() {
	e, err := hqgourlextractor.New(hqgourlextractor.WithScheme())
	if err != nil {
		log.Fatal(err)
	}

	urls := e.Extract("Check these out: ftp://ftp.example.com, https://secure.example.com, and mailto:someone@example.com.")

	fmt.Println(urls)
}

Configure the extractor with functional options. WithScheme() requires a scheme, and WithSchemePattern restricts schemes further:

e, err := hqgourlextractor.New(
	hqgourlextractor.WithSchemePattern(`(?:https?|ftp)://`),
)
if err != nil {
	log.Fatal(err)
}

With no options, relative paths such as /docs/guide match too. Regexp returns the pattern compiled by New for callers who need match indexes. WithRegexp supplies a complete custom pattern and cannot be combined with the generated-pattern options it would ignore.

The pattern building blocks are exported (SchemePattern, KnownSchemePattern, IPv4Pattern, IPv6Pattern, PortPattern, and others) for reuse in custom configurations:

e, err := hqgourlextractor.New(
	hqgourlextractor.WithSchemePattern(hqgourlextractor.KnownSchemePattern),
)
if err != nil {
	log.Fatal(err)
}

Parsing

The parser package extends Go's net/url parsing with a detailed domain breakdown.

package main

import (
	"fmt"
	"log"

	hqgourlparser "github.com/hueristiq/hq-lib-url-go/parser"
)

func main() {
	p := hqgourlparser.New(hqgourlparser.WithDefaultScheme("https"))

	parsed, err := p.Parse("www.example.com/path/file.txt")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("Scheme: %s\n", parsed.Scheme)
	fmt.Printf("Host: %s\n", parsed.Host)
	fmt.Printf("Hostname: %s\n", parsed.Hostname())
	fmt.Printf("Subdomain: %s\n", parsed.Domain.Subdomain)
	fmt.Printf("Second-level domain (SLD): %s\n", parsed.Domain.SecondLevelDomain)
	fmt.Printf("Top-level domain (TLD): %s\n", parsed.Domain.TopLevelDomain)
	fmt.Printf("Path: %s\n", parsed.Path)
}

Hosts are decomposed with the embedded Public Suffix List. IP literals get a nil Domain. A single-label host such as localhost lands wholly in SecondLevelDomain. The most specific matching rule, including private, wildcard, exception, and punycode rules, supplies TopLevelDomain, with the label before it in SecondLevelDomain and the rest in Subdomain. A host that is itself a public suffix, such as co.uk, lands wholly in TopLevelDomain. Unknown TLDs use the implicit * rule, so the last label becomes TopLevelDomain.

Domain.Apex() returns the registrable domain (SLD plus TLD). For google.blogspot.com, both this package and weppos/publicsuffix-go's Domain include the private blogspot.com suffix by default: Apex() is google.blogspot.com, and weppos returns google.blogspot.com. With WithIgnorePrivate(), this package gives SLD="blogspot", TLD="com", and Apex() == "blogspot.com", while weppos with IgnorePrivate: true returns blogspot.com.

Normalization

URL.Normalize rewrites a parsed URL in place into a canonical form. InferScheme guesses a scheme for a host that is missing one: http for local targets, https otherwise.

package main

import (
	"fmt"
	"log"

	hqgourlparser "github.com/hueristiq/hq-lib-url-go/parser"
)

func main() {
	p := hqgourlparser.New()

	parsed, err := p.Parse("https://WWW.EXAMPLE.COM:443/./a/../b/?b=2&a=1")
	if err != nil {
		log.Fatal(err)
	}

	parsed.Normalize()

	fmt.Println(parsed.String())
	fmt.Println(hqgourlparser.InferScheme("localhost:8080"))
	fmt.Println(hqgourlparser.InferScheme("example.com"))
}

Contributing

Contributions are welcome and encouraged! Feel free to submit Pull Requests or report Issues.

A big thank you to all the contributors for your ongoing support!

contributors

Licensing

This package is licensed under the MIT license. You are free to use, modify, and distribute it, as long as you follow the terms of the license. You can find the full license text in the repository - Full MIT license text.

About

A Go package for working with URLs.

Topics

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages